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FOREWORD 


This is the final report of our research on biocybemetic factors 
in huaan perception and aenory which was begun in 1972 at Stanford 
University and **38 supported partially* or fully by the Advanced Research 
Projects Agency of the Departaent of Defense under co^itract 0AHC15>72» 
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project. 

Since its inception, this project has been the result of the 
collaborative efforts of aany individuals. Most of the staff aeabers 
contributed to the writing of this report. 
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Blocybernetic Factors In Human Perception and Memory 


SUMMARY 

The objective of this research project is to develop biocybernetic 
techniques for use in the analysis and development of skills required 
for the enhancement of concrete iraage^^ the '*eidetic" type. The scan 
patterns of the eye during inspection oi scenes are treated as indicators 
of the brain's strategy for the intake of visual information. We 
attempt to determine the features that differentiate visual scan patterns 
associated with superior imagery from scan patterns associated with 
inferior imagery, and simultaneously, to differentiate the EEC features 
correlated with superior imagery from those correlated with j iferior 
imagery. For this purpose, we have designed a closely -coup led man-machine 
system to generate image enhancement and to train the individual to 
exert greater voluntary control over his own imagery. The models for 
EEC signals and saccadic eye movement in the man-machine system have 
been completed. These models are used for monitoring and prediction of 
EEC signals and eye posi t ions. At this moment, all parts but the feedback 
path of tae man-machine system have been implemented. We shall describe 
in this report the details of these models and discuss their usefulness. 

We have assumed that the strategy of an individual with superior 
visual memory is to fixate his eyes at the optimal locations of the 
visual scene at the optimal time instants in relation to EEG. Through 
the EEG model, we have concluded that the EEG signals play the role 
of timing mechanisms for visual information acquisition and processing. 

By using our characterization of scar patterns, we are able to show that 
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the scan pattern is more consistent for individuals with high scores 


obtained from the Marks' Visual*Memory Task than those with lower scores. 
These results tend to confira our assumption and help guide us in 
obtaining a firmer grip of these optimal locations and time instants. We 
havo also developed and implemented a technique for presenting any part 
oi a scene at a specified foveal location at a specified instant of 
time. If the specified locations and time instants are optimal, we 
expect that the image will be enhanced. Prediction techniques have 
been developed for both saccadic eye movements and EEG waveforms so 
that lead times may be provided for the presentation of appropriate 
visual cues (stimuli). We shall present and discuss many of these 
in this report. 
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I. wrmmKTiw 


Tkim project hM boon aiMd ut tiM dovolopMcat and une of biocjter- 
Mttc con^opta nod tochniquon for nnnlysiiig and dereloplnit akliXa that 
arc «^aaoBtial for tbo oniotti^ aaent of co crate Inaoea of the **eidetlc** 
type, % haae cooceotmted on the problas of acbleainc biocybernetic 
espaanioii of viaual Moory by tinlnc a clooely-eoupled Mn-oachioe syatea 
which porfoma mal-tioe oeaitorinc* analyaia. and feedback of spatial 
and tecporal cues that senre as keys to hunaa neaory encoding aid 
recall. There la strong evidence that th.se cues are heavily depended 
on in nenory encoding and retrieval by hunan nervous systems. The closely- 
coupled* nan-nachioe system is used ior the measurement and prediction 
of human mnemonic performance to determine the spatial and temporal 
cues as well as for the control and enhancement of mne«K>nlc skills. 

Such a man-machine system for visual nemory tracking and training has 
been designed and is depicted in Figure 1, This system is capable of 
delivering optimal sequences of sensory stimulation conditionally 
related to eye position and brain stati, and thus to explore systematically 
their relation to visual memory retrieval. 

It is seen from the man-machine system that ve emphasize the real- 
time monitoring and prediction of central nervous activities through 
the EEC sipnals and through tho tracking of eye movement and eye fixations. 
Employing these real-time techniques* «e have attempted to determine 
the tewiporal and spatial cues for human nemory encoding and recall by 
monitoring of brain states and eye positions. Again, tnrough this closely- 
coupled nan-machine system, ve utilize this information to arrange the 
desired coincidences between various brain states, eye positions, and 
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Bye-vovec’Mit aesmircneat 



Response 


Figure 1 Block diagram of the 
closely-coupled man-machine system 
for monitoring and training of visual memory skills 


the delivery of visual stimulation. The visual stimuli have been 
presented on projection screen or oscilloscope display's for binocular 
or monocular viewing. The eye movement was measured wit i the Biometrics' 
Eye Movement Monitor^ Type SC; it is now measured with the Stanford 
Research Institute's (SRI) Comsweet Eye Tracker. The brain states 
have been monitored through the EEC signals* It is expected that 
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greeter control of Image persistence and image dissipation can be 
obtained bj combining these real-time mcmitoring and prediction 

techniques with a feedback scheme to close the control loop as depicted 
in the block diagram of the man-machine system. In other words, we 
strive to use the computer system to supplement and strengths those 
deficiencies in human memory that ordinarily result in image 
dissipation based on the assumption that a superior seen pattern of 
visual inspection that results in a superior memory is more consistent 
and also less probable of natural occurrence than a visual inspection 
strateg> that is less consistent. To summarize, we attempt to steer 
the subject toward improved encoding and decoding strategies for memory 
try using the techniques which have been developed and implemented on 
this can-machine system. 

The closely-coupled man-machine system utilizes computer-'based 
models in the system configuration to predict the kind of Ftimuli which 
should be used in order to produce the desired future responses. The 
EEG signals and the eye-movement measurement are shown as the responses 
which are closely controlled since the computer-based models are designed 
to mimic the actual physiological processes in regard to their stimulus- 
response relations. To be more specific, since the visual cues (stimuli) 
must be presented at appropriate locations in the visual field at the 
right instants of time, monitoring and prediction on real-time basis 
of both eye movement and EEG signals are essential. These models have 
been developed for the purpose of monitoring and prediction. We have 
completed the work on the computer-based models which will be described 
in detail in the following sections. These models have also been 
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implenented on the All of the nan-machine system, except the 

interface electronic devices, can be realized on the POP-15 computer. 

At the close of this project, we have inplanented all but the feedback 
path of the man-machine s^sten. 

In order to determine the visual cues which serve as keys to 
both memory encoding an decoding, we have to characterize the scan 
patterns of observers during the inspection of scenes. With this 
quantitative characterization, we '«ill then be able to differentiate 
quantitatively a superior scan pattern of visual inspection that 
results in a superior memory from an inferior one. This quantitative 
description is intended for use in tne determination of those vital 
visual cues. We have developed and implemented a statistical method 
for the characterization of visual scan patterns. Our experimental 
results have established that observers with good visual memory, as 
scored by Harks* Yisuai-Uemory Task, have a more consistent eye scan 
pattern in quantitative terms chan those with poor visual memory. The 
characterization of the visual scan pattern and the related eye-movement 
measurement techniques will be discussed in Section IV. 

Last, but not least, we will present a technique for the presentation 
of the visual cues (stimuli) at appropriate locations in the visual 
field at the right moments. Since our ultimate goal of this project is 
the visual memory enhancement through biocybemetic techniques, once the 
visual cues are determined, they can then be presented to the observer 
or trainee using our technique. In other words, this computer-implemented 
technique will enable us to deliver visual stimulation to coincide with 
the desired brain states and eye positions since it controls, in real-time 
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tli6 BODocular field of view of an <rt»aerver inspecting outline drawings 
of visual scenes on a gri^hical CRT display. Hie details will be diecussed 
in Section V. 

In Section VI » we shall sunnarize our acconplislments and sake 
concludij^ renarks. Some of these accompllshnents have been published 
or presented in international scientific conferences. The list of 
publications relevant to this research project is included in 
Appendix B. 

In sunnary, we have designed a closely-coupled nan-nachine system 
for the purpose of developing and use of biocybemetic techniques for 
analyzing and developing skills that are essential for the enhancenent 
of concrete images of the "eidetic** type. This man-machine system has 
been iaqilemented except the feedback path, lu this systen, we have 
developed models for EEC signals during visual stizsulation and for 
eye movement during inspection of stationary scenes, respectively. These 
models are then used for the purpose of monitoring and predicticm of 
various brain states and eye positions so that the desired coincidences 
between various brain states, eye positions, and the delivery of 
visual Sixmulation can be arranged. This arrangement of the desired 
coincidences can be carried out by a technique we developed and implemented. 
The electronic interface systems in the man^machine system serve 
a very important role in data acquisition and monitoring of the brain 
states and eye positions. These electronic Interface systems consist 
of EEC recording machines with Grass preamplifiers, eye-covement measuring 
instrument vSRI Eye Tracker) and related interfacing hardware and 
software. Ve shall cover some of the unique features of the interface 
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systeas in Appendix A, This report will conclude with Appendix B 


which includes: (1) 
to this project; (2) 
are to be published; 


a list of the naaes of those who have contributed 
a list of all relevant papers that have been or 
and <3) a list of lectures and talks given. 
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II. AWAUrSIS AND MODEUWG OF EYB MOVKimiTS 
DURING SCENE INSPBCTION 


In the closely-coupled Ban-«achine systra depicted in Figure 1, 
we have two boxes » each labeled as conputer-based model; one is for eye 
movement and the other is for KBG signals. These coeq[>uter-based models 
are used for monitoring '^nd prediction of eye positions and various brain 
states, respectively. The monitoring of the eye positions is necessary 
for the determination of the spatial cues that serve as keys to vi&ual 
memory encoding and retrieval and the prediction of eye position is 
essential for the presentation of these cues at the desired locations 
of the visual field. In this section, we shall discuss the development 
of such a model for eye movements during inspection of two-dimensional 
scenes . 

1. An Algorithm for Autoaiatic Identification of Fixations and Saccades 

Eye movements during inspection of two-dimensional stationary scenes 
consist mainly of two components: fixations and saccades. The first step 

in our analysis of eye movements is to senarate the data into these two 
components. Ne need to determine the beginning and end points of the 
fixations and saccades. This helps us to determine the sequence of 
fixations during a volitional scan of a two-dimensional scene. The 
separation of saccades from fixations Is useful for the modeling 
to be described later. In the past, the delineation of fixations 
and saccades and the scoring of the fixation sequences has been done 
by hand. We have developed an algorithm which does this automatically 
and can thus reduce large amounts of ‘ iw eye-movement data to a sequence 
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of fixation points very quickly. With nodi float ions, the algorittan can 
perfom In real tine. In addition, it gives us an objective nethod of 
data analysis as opposed to the earlier nethods. 

The detection of saccade onset is essentially a problen of signal 
detection. We can consider the movements during fixation as noise and 


the saccade as the signal. At any instant, we compute the quantities 


*. ■ l*F - £Vll 


!», - s Vil 

i=l 


where saaq^led values of the horizontal and vertical c<xq>onents 

of the raw data; is the location of the fixation; and ((u 1^ "" 

a weighting function of length k « The algorithm indicates the onset 
of a saccade at t = 0 if either R > T or R > T , where T is a 

* y 

preset threshcIJ value; otherwise, the weighting window is moved 

forward. The threshold T is a parameter of the algorithm which 
depends on the noise level in the measurement, the size of the smallest 
saccade that has to be detected, the amount of delay (k) allowed before 
detection, etc. This approach is essentially equivalent to using a 
matched filter since the weights are determined using the saccade 

model to be described later. Based on this model, we can determine the 
error probabilities for different size saccades and set T accordingly. 

We use a slightly different approach to determine fixations since 
the end of a saccade is usually much slower than its onset. Now we 
compute 

( i i 

“ = ^ - s g^xj < T A ly - E g y.| < T , 1 S j i i 

( j i=l j i=l ^ ® 

where [g ) is an empirically determined weighting functicxi and T is 
i 8 

a threshold value depending on the noise variance. In effect, we are 
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computing; the number of points in a sample window of length i which 

falls within a square of size T around the window weighted mean 

s 

algorithm moves to the fixation state if n 

exceeds a preset value n (< j() ; otherwise, the sample window is moved 

s 

forward. In practice, a simple averaging function is found to be adequate 
for (g^) . 

We have written a program to ir’plement this algorithm. The program 
reads raw eye*data from the magtape and uses the algorithm to determine 
the fixation points. These are plotted and numbered. The program also 
outputs the starting and ending times and the standard deviations of 
each of the fixations. An example is shown in Figures 2, 3, and 4. 

Figure 2 shows the raw eye-movement data. Figure 3 shows the sequence 
of fixation points obtained from this algorithm, and Figure 4 shows the 
fixation points superimposed on the actual stimulus used. It is clear 
that a considerable data reduction has been achieved and the result is 

easier to understanc and analyze. Other advantages of this algorithm 
include easy adaptation to different noise levels and/or measuring 
instruments by changing some parameters, and a quantitative estimation 
of error probabilities. In fact, the algorithm has been used quite 
successfully during the course of this project cn data obtained from both 
the biometrics unit and the SRI eye tracker by simply changing the 
thresholds. 

2. Modeling of Saccadic Eye Movement 

The next step in the analysis of eye movements is the investigation 
of saccadic movement* Wc need to be able to monitor and predict the 
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Figure 2 Raw eye movements for 
a 20-second scan. Scale = 2.5 deg/inch 
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Figure 3 Fixation sequence for the scan in figure 2 
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saccade duration and length in order to present visual cues (stimuli) 
at the appropriate locations in the visual field at the desired moments. 
Thus, the detailed dynamics of the saccadic movement are of interest to 
us. Hence, our approach is first to develop an aporopriate model which 
embodies the Dynamic characteristics of the saccadic system. This model 
is then utilii.ed for monitoring and prediction of the saccadic '•-.cveroent. 
The human saccadic eye-movement control system has been of great 
interest to researchers in neurophysiology and bioengineering. The 
general shapes of the position, velocity, and acceleration curves of the 
eye movement have been reported extensively. We have attempted to fit 
the Tiodel responses tc empirical data by classical least-squares 
techniquej. This enables us to estimate certain parameters of the model. 
These parameters which will be used for predicting the final eye positions 
have only been determined by indirect means in the reported models. We 
have also made important modifications to the model so that its output 
will accurately characterize the observed responses. Wo shall describe 
and discuss the model modifications, the parameter estimates, the curve- 
fittin^; results, and the use in m.aitoring and prediction. 

ror the present, we are concerned with horizontal eye movements 
only. For this purpose, an experiment for the study oj ho^'izontal 
saccades has been designed and used. A row of LEDs cqu i? tantly 
located, only one of which is lighted at any given time, is used as 
stimulus. We will refer to the leftmost position as the rest position 
and the o*^hcr six as the target positions. The target signal is supplied 
by a digital computer. The subject is instructed to follow the light 
spot, which repeatedly .jumps from the rest position to one of the target 
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positions and back. The successive target positions are chosen from a 
table of random permutations tc uvold anticipation by the subjects. An 
interval of at least 1 second is provided between each position change. 

Eye movements are recorded and stored as described elsewhere in this 
report , 

Using the algorithm described above, we first mark the starting points 
of the saccades. Then the saccade position and velocity are plotted 
and equal length saccades are averaged with each sample lined up at the 
start. As an example, we show position and velocity plots of a typical 
saccade in Figure 5 and the averaged plots from 25 saccades of one 
subject in Figure 6. The saccades observed agree with those reported 
by other researchers in all aspects such as duration, overshoot, 
response delay , etc . 

In order to monitor and predict the saccadic eye movements, we intend 
to fit the model output to the saccade position data by estimating the 
model parameters from the data. The endpoint cf the saccade can then be 
predicted from the model output by using these parameters. 

Let the liorizontal position y at time after vhe onset of 

the saccade be 

^ * ^2 * * * * * * ^i ^ 

where a ,a , .,,,a are the model parameters and n, is ine noise 

X A K X 

term which represents the errors introduced by the instrument, the data 
collection errors, the inherent random nat* of the biological system, 
etc. We shall use the least-squares estimation to determine a ,a , 

X A 


14 



T r 

if 


2f 


3f 


4f 


50 

TIME 


00 


70 


Kigure 5 Typical posit^oa and 
velocity plots of a 15'^ . ccade 


80 


90 t00 


15 



X-VEL X-POS 



TIME 



TIME 


Figure 6 Averaged plots iron 
25 saccades of 10^ amplitude 
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. They are deterained by ainialzing the aeao-square error 
S = ' e(aj.a2,...aj^;t^)j 

where n is the nuaber of saaples observed; i.e., by aol' ..ig t\e set of 
equatiODS 


ds 



= 0 


as 



= 0 


as 



0 . 


( 2 ) 


In the case when 6 is a linear function of the parameters * * * * 

Eq« (2) is reduced to a set of linear equations which can be solved to 
obtain the parameter estimates. If 0 is nonlinear, then an iterative 
process has to be applied in order to solve for • 

If we assume the noise n to be white Gaussian, the least squares 

X 

estimiles are known to be identical to the maximum likelihood estimates 
which are unbiased, consistent, and asymptotically efficient. With the 
above assumption, we can also estimate the variar js of th > parameters 
and use X^-test and t-test for goodness of fit. This gives us a 
quantitative measure of the goodness of our model. 

The functional form for 0 is determined by the particular model 
used. It undoubtedly relates to the dynamics of the eye-movement 
system. We have chosen to use Robinson’s model for its simplicity. 

This model is block diagramed in Figure 7. 
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Nl^Lb Plant 


Figure 7 A model character ^ ring 
dynamic properties of saccadic 
eye movement 


Here we have shown the dynamic part of the model only; the controller 

part is left out. The input to the system is a pulse of height h and 

width T • This produced a saccade of amplitude equal to the area under 

the pulse; i.e,, hx • N1 is the neural Integrator and ULF the medial 

longitudinal fasciculus which is a feed forward path in parallel w^th 

the integrator and provides a lead network to compensate for the plant 

lag. R represents the oculomotoneuron firing rate. For a pulse input, 

we see that R is a pulse step as desired. The plant is a second- 

order over-damped system with time constants T^ and T^ • Robinson 

chose the MLl gain • However, he suggested that in order to 

produce the observed overshoots and undershoots T > T and T < T , 

o X *5 X 

respectively, should be used. For our purpose, we have decided to 
retain T as a separate parameter. The output 0 is the eye position. 
The cony)lete transfer function is then 


V 


T3H + 1 


(T^s+lHT^s+l) 


For the assumed pulse input, the output 0 (t) becomes 
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The po«iticm and velocity curves given by the above for h • 5 dec/as 
and T s 20 ns corresponding to a 10 " saccade are shown in Figure 8* 

The values of the other paraaeters used are T^ = 150 as and T^ ^ 7 u. 
(These values are suggested by Robinson.) Curves are shown for T^ = 190 as, 
150 as, and 110 as. Note the overshoot and undershoot for > 

and T < T , respectively. 

Note that there are four unknown paraaeters, » ^2 * * 

For estiaation, we have to solve £q. (2) for k = 4 • However, froa the 

theoretical evaluation of the parameter variances, we found that the 

estimate of T^ will have a very large variance. Thus, the estiaate 

will not be very meaningful and hence, a fixed value of T^ = 150 ms 

will be used. As will be seen later, this does no: affect the 

results significantly. We have also found that, for reasons of nuaerical 

calculation, it is better to estimate T^/T th^ T itself. Thus, 

•j' 1 3 

three parameters a = h , a = T , and a = T /T were estimated by 

X ^ ^ o X 

using the data from t^ to ^ ^ * where t^ is the time of the onset 
of the saccade and t is the input pulse width which is estimated from 
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the saccade velocity by noting that the velocity reaches a ■axiaua at 

t^ -t- 1 as shown in Figure 8. The estiaated parameters are then 

substituted into Eq. (3) to obtain the complete saccade curve and the 

prediction oi the final position is made simply bX emtrapolatioa. 

Typical results Irom the parameter estimation and prediction 

schemes are shown in Figures 9 and 10. The model output matches the 

actual data quite closely. The goodness of fit has been tested usiny the 
2 

X •test on the sum of the squares of the residuals, S , and the t-test 

on the individual residuals at significance level P - .05. F..He that 

the estimates of T^ are 10.9 ns and 13.6 ns as <H>posed to the value 

suggested by Robinson. This result is typical and the mean value of 

T^ is about 13.0 ns, although it varies between subjects. It is, 

however, significantly different from 7 ns. 7 /T >1 is observed in 

O X 

Bost cases » although the difference is not significant as far as 
the final value prediction is concerned. The values of h and t 
depend on the saccade size which implies that the size is controlled 
by both the amplitude and the width of the pulse. Tlie exact relationship 
IS being investigated. 

In most responses » an overshoot of the final position was observed. 

This overshoot is not produced by the model, as can be seen in Figures 

9 and 10. Even if we vary T • the overshoot could not be produced. 

3 

This is because fo* overshoot decays with a time constant 

•5 X 

T^ , whereas the actual overshoot decays much faster (at about the same 
rate as the initial rise). In addition, we could not fit the model 
response beyond t to the observed data within the chose i significance 
level P - .05. This indicates that the model must be modified to 
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produce the right response. The fact that the overshoot decays with a 
time constant of about T and the input to the agonist and the antagonist 

A 

muscles used by Clark and Stark suggests that the input to the model 
consists of a combination of pulses rather than a single pulse. A 
study of the velocity curve in Figure 6 also suggests that a positive 
pulse followed by a negative pulse can produce the desired output. 

Thus, our model input is changed to two pulses as shown below: 



pulses with saccade amplitude equal to 



h. t. . 
I 1 


In practice 9 we found that no more than three pulses were ever necessary. 

The results of this modification are seen in Figures 11 and 12. 

For all multi-pulse input fits, T = T = 150 ms was assumed. Figure 

o 1 

11 shows the results of estimation from a single pulse input. Figure 12 
shows the same saccade with the model output from a two-rulse input 
keeping other parameters the same as before. The fit in the second 
case is clearly much better. Figure 12 also shows the fit of the 
velocity curves from the data and from the model. Thus, excellent fits 
are obtained to both the position and velocity curves with this model. 
Figure 13 shows a much more dramatic result where a three-pulse input 
to the model is used. The Justification for using such an input can be 
clearly seen from the velocity curve which first rises to a maximum 
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Figure 12 Actual and model simulated 
saccade position and velocity 
with two^pulse input 
t^ = 20 ma, = 150 ms« = 13.1 ms, 

= 150 mS| = 18 ms, = .36 deg/ms, 

Tg = 27 ms, hg = -.05 deg/ms 
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value » then approaches a steady positive value | followed by a rapid 
fall to a negative maximum and decay to zero* This identifie:? four 
distinct regions, suggesting four different input levels or three p^^ises 
(one level being zero)* In fact, for the fitting, the pulse widths are 
identified by the instants where the velocity signal changes character* 
This would he easier to identify on a more noise-free system where the 
second derivative or acceleration can be observod* However, it is 
clear that the input is more complex than a single pulse* At present, 
we are trying to investigate the relationships between the heights and 
widths of the various input levels for different saccade lengths for the 
purpose of prediction* Similar responses are observered for equal 
length saccades which indicates that some identifiable relations exist* 

We are also trying to extend the results to vertical ' 'Oblique saccades* 

The above approach is useful fcr prediction of the final position 
of the saccade, given a part of the saccade; l*e«, we can use it for 

prediction only rtter a certain time has elapsed since the onset of 

the saccade. However, we also need to know when a saccadt. will occur 
in response to a stimulus. Normally, there is a delay of between 150 
ms to 250 ms from the presentation of the stimulus to the saccadic 
response. If we can predict t!:3 actual delay, we can predict the 
course of the eye Liovements following the presentation of the stimulus 
by usin^ this delay in conjunction with the above model. We decided 
to use information in the E£G alpha rhythm to predict the onset of the 
saccade* Gaarder et al. have shown that fixation saccades are initiated 
during a particular phase of the alpha* This is in accordance with 
Wiener's hypothesis that the alpha cycle serves as a clock which 
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provides tiains for tOe data gathering and processing portions of the 
visual sjstea. This result vas obtained froai nicrosaccades occurring 
during fijcati'^« We want to deteraine if it applies for the large 
saccadic aoveaentr as veil. If so. ve can use an EBG-alpha-wave 
predictive scheae to predict the phase of the vavefora and then use 
this inforaation for estxaating the onset of the saccade. 

The saae ezpcriaent as described earlier for studying horizontal 
saccades and in addition to record siaultaneously the CBG aignala coll 
as described in subsequent sections has been perforaed« The EEC was 
filtered in the alpha range. Two aethods were used to deteraine the 
relation between the saccade onset aid the alpha phase. Hrst, the 

quadrant of alpha during which the saccade started was deterained. 

2 

The X -test was used to test the distribution of saccades in each of the 
four quadrants, which was found to be significantly different (p < .05) 
fron a unifom distribution. This test is identical to the one used by 
Gaarder et al. In addition, the alpha was averaged with the saccade 
onsets lined up. Results indicate that there is indeed a preferable 
phase of the alpha rhythm .r onset of the saccade. This fact can be 
exploited for the prediction of the onset of s<iccades if a scheme can 
be found for predicting the EEG alpha waveforms; espeically, their 


phasic variations. Such a method will be described in the next section 



III. MODEUNG OF KEG SIGKAI^ 

DURIWG VISUAL gTlIKJLATlOW AMP PREDICTIOW 
OF EEC ALPHA WAVEFCHniS 

In order to determine the teaporal cues that are Ivportant in visual 
memory j it is necessary to understand the relation between the tenporal 
structure of visual perception and the EBG alpha cycle. To this end, 
ve have investigated the changes produced in the EBG signal by sinple 
visual stiniili. In particular, we are nodeling and investigating the 
various EEG entraiment phenonena caused by periodic photic stinuli 
and the nhasic changes in EEG due to photic stisnili. This will enhance 
our knowledge in several areas: it enables us to unify several seeningly 

different EBG phenowena under one principle, prc /ides a model for the 
phase-dependency of stimulus efficiency, and enables us to study the 
phase-dependencies of onset times of saccades. A number of researchers 
have suggested that the alpha cycle is phase locked with a clock signal 
which provides the time base for a sampled-data system consisting of 
the eye, its control mechanism, and the visual cortex. The evidence 
for this has been accumulated slowly; it is known that visual perception 
times and reaction times are on the order of one alpha cycle and dependent 
upon the phase of the stimulus. In addition, some recent work has 
shown that a saccade is most easily initiated on certain phases of the 
alpha rhythm, which coupled with the fact that vision is blocked 
during a saccade, suggests that the alpha cycle delineates visual data 
sampling and processing periods. 

The above discuo^lon outlines the motivation for modeling the EEG 
signal during simple visual stimulation; the modeling efforts to date 
have been based on a simple nonlinear oscillator which is amenable to 
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analysis and sinulation 


1 . A Nonlinear Matheaatical Model for Entrainaent of EEC Signals by 

Periodic Photic Stimulation 

The ROdel is a van der Pol oscillator representing the behavior of 
a subject's ££G during periodic visual stimulation by a stroboscope or 
a set of l£D goggles. The stimuli are delivered for 20 seconds at 
some fixed frequency followed by a 10-second stimulus-off period. This 
sequence is repeated for each desired stimulus frequency. 

The van der Pol oscillator can be represented by 

X -^I(l - x^)x + = .^(t) 

where x(t) denotes the EEC signal; is the unstimulated alpha 

frequency; E(t) is the external excitation (stimulus); and p. is the 
nonlinear coupling coefficient. 

When E(t) is a sinusoid, a first approximation to the solution 
x(t) can be made by a classical technique, such as harmonic balancing. 
The possible solutions can be classified in the following way when 
E(t) = E^sin(-j^t + E^): 

(1) natural oscillation (E^ is small): 

x(t) = A sin(^^t + <P) ; 

(2) harmonic entrainment 

x(t) = A(t) sin(-^j^t -r 0(t)); 

(3) mth-order subharmonic entrainment = ^^) : 

x(t) = A(t) sin 

(4) nth-order superharmonic entrainment * nu.*^): 

x(t) = A(t) sin(nu.^t + ^(t)); 


t ^ 0(t), 
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(G) coabined frequency oeclllations (all other casee): 

«(t) = A^(t) 8in(u^t + + A^(t) sin(h«^t + 

Soae Initial data have been collected with a set of specially 
constructed L£D goggles which provide a sinusoidally nodulated intensity, 
and the results agree well with the above solution classes. However, 
since the esperiwental stinulus id often a train of flashes, we have 
developed an analysis technique for approsinating the solution of the 
van der Pol equation when E(t) is a pulse train. 

We assume that the unperturbed oscillation (E(t) = 0) is correctly 
represented by 

*<t) = Sq sin e(t) 

where the angular displacement can be written in tezms of frequency and 
phase: 

e(t) = (Jj^t + ♦(t) . 

We wish to examine the result of E(t) being a series of impulses of 
strength q at a frequency . 

Following a method due to Blacquiere, we can assess the perturbation 

caused by one impulse. At an angular displacement 

0 = w t + a 

n On 

we apply the nth impulse. Its effect will be seen directly in S(t) , 
causing a small step change 

t -hit 

/*” 2 
A* = / x(T)di = Q 

^n 


over tine At . However, x will be continuous, so 
To a first approximation, we can expand Ax and 
Ax = AS sin 0^ XqA* cos 0^ = 0 


AX = 0 . 

AX as below: 
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2 

AS s 90S d ~ s u A^sin 9 s qu 

“0 'n 0 0 n 0 

where as represeDts the chasse in aaplitude and a^ repxeaents the 
change la phase of the systen* Solving the above equations simultan- 
eously for AS and A* » «o obtain 

Aa s qu^ cos 0 ^ . 

and 

n 

A» sin e . 

*0 “ 

In Figure I4t «e deaonstrate the aaplitude and phase perturbations as 
a result of an iiqralse falling on angular displaceaent 0 



The unperturbed solution would have followed the dashed line, and the 

perturbed solution follows the solid line. 

In order to deal with a series of these inpulses, as will be 

assuned to be negligible. This is reasonable since a more detailed 

analysis shows that the aaplitude change decays exponentially with 

tine. Let T s and T. = be the alpha and stiaulua periods, 

0 1 u). 
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respectively. Then referring again to Figure 14 > ve see that apply! ■:$ 
the nth pulse at dxs^laceaent 9^ results in a nee displaceaent 
9 ^ + • Since the pulses are spaced seconds apart, the 

(n-(-l)th poise will arrive at displaceaent 




Cler.rly, for entraioaent to occur, «e must hare the congruence relation 


e H e (aodulo ) 
n+1 n 0 

or 

e + A^(e ) T, H 0 (nodulo T^) 
n n 1 n o 


Hence 

A9(e ) + T = k T , k = 0, 1, 2, ... . 

n 1 u 

When k s 0 , we have the unlikely situation of resetting the phase of 

oscillation; the required phase change / is very large. We will 

n 

not consider this case here. 

For k = 1 , the solution is called harmonic entrainment, and for 
k ^ 2 , it is called k^^-order superhamonic entrainment, where k 
represents the number of alpha periods elapsed per stimulus impulse. It 
is easy to show that this is stable near 



mod 

0 


0 . 


one would expect from the form of • 

In a similar fashion, for sebhamonic entrainment to occur. 


®n-h« ^ ®n ”" 0 ^ 
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or 


d4b-1 

e. + E +«nr, se (aoduio t j 

O . 1 1 n O 


Hence » 

n+n-1 

S A^(0j) + ■rt’i = • 

th 

Vlien ■ - I ve have hanonic entraiiuient again, and for n ^ 2 » n - 
order subhamcmic entrainsent occurs. The stability of this solution 
is currently being investigated, but it appears that for ■ = 2 a 
snail stable range exists. 

In sunnary, when the stimilus frequency is near the alpha frequency, 
hdmonic entrainent is possible. When the alpha Trequency is near an 
integer Multiple of the stimulus frequency, superhamonic enCralnnent 
is possible. And when the stinulus frequency is near an integer Multiple 
of the alpha frequency, subhamonic ent raiment is possible. 

When the stiwilus frequency is not ir one of the above ranges, 
coBbined frequency oscillations exist. Thus, ve have the sane five 
solution classes for both sinusoidal and iMpulsive stimuli. The major 
differences lie in the amplitudes required to produce the various effects, 
and in the fact that the impulsive stimulus will produce integer harmonics 
of its fundamental frequency due to its inherent harmonic content. 

Results obtained from EEC data on the model are described and compared 
in the next section. 


2. Model Simulation 

It is necessary to simulate the nonlinear oscillator model to 
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obtain reasonably accurate solutions since nonlinear analysis is by 
nature approximate. We are using a variable o:*der differential equation 
solver (DVDQ) which has an Adans-Faulkner predictor and an Adams- 
Moulton corrector. It provides for up to twentieth order interpolation 
of output values positioned independently of the changing stepsize, 
which greatly facilitates signal processing of the simulated solution 
by allowing a constant sample oeriod. 

The sine wave stimulus is handled in a straightforward fashion, but 

the impulse train is treated specially. Since it is known analytically 

2 

that the exact change in x(t) is (by integration) at the tine 

an impulse is called for, the state vector is perturbed and the integration 
restarted. The management of impulse arrival and sample output is 
performed by a discrete event*queuing scheduler on a priority basis; 
impulse arrival has priority over all events to avoid frequency shifts 
in the impulse train. Error propagation as a result of restarting 
the integration is linear with the requested error tolerance and 
adequately small. 

Figures 15 and 16 show the phase and aaq)litude perturbations caused 
by a single flash landing on phases of 0 , tc/2 , n , and 3t(/2 radians. 

The simulations match the first order aporoximations developed in the 
previous section very well. These perturbations produce appropriate 
entrainment phenomena, as seen in the spectra of Figures 17 and 18; 
there is one spectrum for each stimulus frequency, representing 5.12 
seconds of impulse driven simulation at a 100-Hz sampling rate. The 
five phenomena predicted in the previous section can be clearly seen; 
viz., harmonic entrainment, subharmonic entrainment, superharmooic 
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Figure 16 Simulated EEG perturbations resulting 
from flash at phases oi it and 3 n/ 2 radians 
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Figure 17 Simulated ££G spectra for flash 
frequencies from #5 Hz to 12.5 Hz 
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e 18 Simulated EEC spectra for flash 
frequencies from 13 Hz to 25 Hz 


40 


entrainment » c<»abined frequ«»ncy oscillations, and stimulus harmonics. 
These effects can again be seen in Figures 19 and 20 which are spectra 
from 5.12 seconds of sine wave driven simulations. These spectra will 
be compared with those obtained from experimental data in a section 
below; the next sect it n discusses data collection and processing. 


3. Data Collection and Processing 


The data collected to test the model have been obtained from a 
very simple experimental paradigm. Subjects are presented a series of 
stroboscope flashes or sinusoidally modulated intensities through closed 
eyelids; the stimulus is delivered for 20 seconds at some fixed frequency, 
followed by a 10-second rest period. This sequence is repeated for each 
desired frequency, typically .5 Hz through 25 Hz in .5 Hz steps. The 
frequency pattern can be selected arbitrarily. The K£C signals are 
obtained from left and right occipital electrodes referenced to yoked 
earlobes; the ground electrode is on the mastoid. The Grass preamp 
bandpasses the signal between 1 Hz and 300 Hz. and the A/D analog 
prefilter is a 6-pole Bessel with a lOO Hz lowpass cutoff frequency. 
Digitizing is at 1000 Hz. and subsequent filteiing is done by a trans- 
versal filter to avoid further phase distortion. Power spectra are 
' ^uted by Fast Fourier Transform! on 5.12 seconds of the data, forming 
the power spectrum and then averaging three such spectra 4.0 seconds 
after the start of the stimulus. ^ » x y ^d\xre substantially reduces 
the noise, which is further reduced for viewing a^st etics by the non- 
causal recursive filter 


H(z) 


+ 2z + 1 
4z 
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Figure 19 Simulated EEC spectra for sine 
frequencies from .5 Hz to 12.5 Hz 
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Figure 2') Simulated EEC spectra for sine 
frequencies from 13 Hz to 25 Hz 


13 


The resulting ^>ectra maj be stqierinposed with hidden lines renewed by 
classical graphics techniques; a snail anount of infomation nust be 
sawed after each spectnin is plotted. 

The stroboscopic stimuli are produced by a Grass PS-2 strobe lamp 
a >oend-vroo^ enclosure. It is driwen by a relay contact. The 
siuusoidal iy nodulated intensity stimuli are produced by a pair of 
!>.enispher^cal goggles, each containing ten ia>s, driwen by a current 
socrce controlled by a D/A converter on the PI^-15. The woltage supplied 
15 proportional to eap (sin cj^t) since tbe l£D amplitude is linear 
with current and sinusoidal intensity is desired. Both stimuli are 
deliwered through closed eyelids to ensure a uniform wisual field. 

Coiu>ari8on of Model Simulations and Eaq>erimental Data 

The comparison of model simulated results with experimental results 
is subject to difficulties arising from the noise in the data and the 
random nature of much of the EEC signal. The alpha frequency Is not 
fixed and is in fact nonstationary; it is influenced by the subject's 
level of alertness and other uncontrollable factors. The response 
of the EEC to a stimulus is only measurable in the average, since 
individual responses are masked or corrupted by noise. Averaging can 
also mask certain characteristics of the response, depend! ij upon the 
scheme used. 

The best comparison of model-produced results and data lor entrainment 
phenomena is a comparison of their power spectra. This allows the data 
spec c rum to be re!*atively free of noise by averaging successive spectra 
from the same stimulus epoch. Examples of stroboscopically stimulated 
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data spectra say be seen in Figures 21 through 24, Kote the excellent 
natch overall with the simulated spectra discussed previously, 
last subject has virtually no alpha rhythSt tmt exhibits all of the 
entrainaent phenomena predicted, so the model accounts surprisingly well 
for extreme cases. Fi jres 25 and 26 show spectra for a sinusoidally 
stimulated subject. Note the reduction in harmonic amplitude as predicted 
by the simulation. Some detailed comparisons of each type of phenomena 
follow. Figure 27 shows the unstimulated case with the model parameters 
adjusted to match the subject (records were selected from Figures 21 
and 22). Figure 28 illustrates harmonic entrainment, with subham<xiic 
and superharmonic entrainaient treated in Figures 29 and 30. Figure 31 
is an example of combined frequency oscillation* The mod^l simulations 
agree veil with the data. Additional comparisons may be found in 
previous reports. 

Further comparisons are in progress, especially the compariscm of 
phase and ai^litude changes induced by the stimuli. Preliminary results 
indicate that the model predictions are reasonable first order approxima- 
tions to the data, but that revisions in the model will likely be 
necessary to fully account for the observed phenomena. The excellence 
of match in the frequency domain bears out the goodness of the phase 
shitt predictions for snail phase shifts, but it appears that larger 
shifts are occurring than predicted for certain cases. 

In sum, a nonlinear model for the behavior of the EEC during visual 
stimulation has been analyzed and compared with actual EEC data. The 
model accounts for several phenomena well in a unified fashion, and 
suggests other phenomena of Interest. The model specifies a trigonometric 
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Figure 21 EEC spectra fron subject L. .‘1» for 
flash frequencies ,5 Hz to 12.5 H* 
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Figure 22 EEC spectra from subject L. M, for 
flssh frequencies 13 Hz to 2C riz 
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Figure 23 EEG spectra from subject M, for 
frequencies .S Hr to 12*5 FFr 
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Figure 24 ££G spectra from subject H» for 

flash frequencies 13 Hz to 25 Hz 
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Figure 25 KKG spectra from subject K, S* for 
sine frequencies .5 Hz to 12.5 Hz 
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Figure 2(^ BEG spectra from subject for 

sine frequencies 13 Hz to 25 Hz 
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Fiirure 27 Actual and simulated ££G spectra 
In the absence of stimulation 
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Figure 29 Actual and simulated EEC spectra 
during flash stimulation of 21.5 Hz 
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fora for the excitability of the EEC as a function of phase, which 
explains well the various entrainaent phenoaena seen in EBG data. 

Detailed phase analyses are in progress which indicate that the aodel 
prediction is good to a first degree, but needs soae revision. One 
possibility at this point is the existence of an excitability function 
at haj f the alpha frequency. The phase-dependency of the aodel response 
has considerable bearing on the phase-dependency of saccade onset. 

5. Prediction of EEC Alpha Waveforas 

In order to present the visual stinuli to coincide with the most 
favorable brain state and to predict the onset of saccadic eye moveaent, 
it is necessary to have the capability to predict the EEG alpha waveforas, 
in particular, their phasic features. In conjunction with the EEG aodel, 
we have developed and iapleaented an EEG-vavefora predictive scheme by 
using aut egressive processes. The developaent and implement at ion of 
this scheme will be aescribed in detail. Our discussior. will be 
centered around the modeling of EEG data by an autoregressive process 
and its use for forecasting. 

The iitting of time-series models to EEf signals has been treated 
in the literature. They have been specifically applied to c£C analysis 
in tne following areas: use for EEG spectral analysis; use of the 
mixed mode] to define certain parameters for describing the stationary 
parts of the EEG signal; and use of the one-step ahead prediction 
error ^o compare two models. Our interest lies in the nrediction 

of the alpha activity in the EEG for a longer lead time. 

Although autoregresriv? ore cesses have been used icr prediction 
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of tiae series in general, they have n^t been used for prediction of 
EEG data. In those general cases, the signal or tiae series is assuaed 
to be stationary. EEG signals are, in general, nonstationary. For 
expedience, they aay be considered p iecevise-s tat ionary ; i.e., stationary 
over short intervals of tiae. Hence, in our application, ve are 
restricted to liaited saaple lengths. This restriction creates certain 
probleas which ve shall discuss later. 

To aodel a discrete tiae series X. , i = 1, 2, by an auto- 

regressive process of order p , ve aay write 


X 

t 


a X + a X -h 
1 t-1 2 t-2 


^ a X 

P t-p 


e 


t 


where ^ 2 ire the parameters of the process to be estivated 

12 p 

and is a zcro-aean white Gaussiar« noise. It is generally assumed 

that the process X^ has zerx> mean; otherwise, the aeac is subtracted 
from the entire series. In general, the autocorrelation tunction of an 
autor ^gressive process consists of two ccaponents; viz.; decaying 
exponential and damped sinusoidal waveforms. It has been recognized 
that the real roots of the characteristic equation of the autoregressive 


process, i.e,, 

..2 p 

1 - a B - a R -aB=0 , 

12 p 

give rise to the decaying expoentials and its complex -conjugate-root 
pairs are respc/.sibic for the damped sinusoids in the autocorrelation 
function. We show in Figure 32 the autocorrelation function of an EEG 
sample filtere in the alpha range. It is seen that the EEG autocorrelation 
function may be 11 approximated by a sum of decaying exponentials and 
damped sinusoMs. In I-igure 32 , we superimpose the autocorrelation 
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Figure 32 Autocorrelagrams : solid 

curve from a typical EEC alpha waveform and 
dots from an autoregressive process fitted 
to the data 
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function of an autoregressiv vrocesa fitted to EEG data to shoe our 
assertion* This leads naturally to the teaptation of Modeling EEG signals 
by autoregressive processes* Since autocorrelation function is not 
a unique procesSt there are other processes which will give rise to 
the saae autocorrelation function. We chose an autoregressive process 
based on its sisplicity in iaplesientation. 

To accoaq>lish our ain iu prediction, we have broken down the problea 
into two parts: 

(1) Estiaation of the autoregressive paraaeters; 

(2) Generation of "good** forecasts according to soaie criteria 

using the above estiaation. 

Since the EEG process is considered piecewise stationary, the paraaieter 
estimates have to be updated continuously. Thus, a scheme is needed 
for updating the parameter estimates as new data arrive. For estimation 
of the parameters, we have used the least-squares principle. These 
estimated parameters are then used to generate the minimum mean-square 
error forecasts for a specified lead time. We developed a new modified 
scheme for v'^is purpose. 

To furnish the basis for our modified scheme, ve shall describe a 
scheme ordinarily used. The development and implementation of a modified 
scheme to suit our needs will then be presented. Results have been 
obtained by applying our scheme to real EEG data. We will present these 
results to show the utility of our prediction method. 

We will concentrate our discussion to a parameter estimation 
method. Later we will present our modifications to this method in 
order to improve the performance for our purpose. Let us consider 
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the p -order process 

*t “ * Vt-2 ^ Vt-p * *t 

vhere e^ is white Gaussian noise* The order of the autoregressive 

process ts estimated by successively fitting BCdels with higher order 

to the data and determining when the last parameter a becomes small 

P 

enough* 

The least-square estimates of the parameters can be shown to satisfy 
the following set of linear equations 


*lS2 *2*^23 ••• V2.P+I = “^12 

’*1^23 ®2^3 ^ “pS.P+l " "^13 




+ a Q 


1 2,p+l 2^3 , ptI 


+ . • . + q Q 


p p+l.p+l 


= -Q 


l.P+l 


where the Q 's are given by 
ij 


” 11 ' ^ 
1=1 


‘iJ 




X , + XX 


2 j-rl 


+ • • • + X 


n-j+1 


and 


Q = X X + X 
i } 


i j i+1 j+1 
1 f 1 . j f 1 


+ .** + X ..X . 

n-j-rl n-i-hl 


Knowing solve the set of ^f^Mations to obtain the least- 

square estimates of the parameters a. » a , sl . The least- 
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Previous discussions furnish us a numerical procedure for estimation of 


the parameters 


9 • • • 9 


and 


in the least-square sense from 


the data (X,» X., X ) « Based on this procedure, one can model 
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a weakly stationary process by a p^*^-order autoregressive process. 

Since EEC data exhibit nonstationarityt this Bethod has to be aodified 
for our problem. Besides, our interest is in the use of the paraaeter 
for prediction rather than nodeling. 

Having obtained the estlauites of the parameters, we turn our 
attention to the problem of prediction. We need a scheme whereby the 
prediction can be made for a certain lead time and then when a new 
data point becomes available, the forecast can be updated without 
repeating the original process all over agcin. 

We Will consider here minimum mean-square forecasts only. Denote 
the estimation of obtained at tine t by • At time 

t , the values of the samples up to time t are known. The minimum 
mean-square error forecast is given by (using the projection theorem): 

^t+i|t = Vt+f|t Vt+je-2|t + ••• + %*t+f-p|t ’ 

which provides the forecasts in the form of a difference equation. 

Given the sampl° values X , X X , we can progressively 

calculate , ^t-r2jt * ••• using the above equation. Using 

this procedure, we can generate forecasts for any lead tine utilizing 
the parameters estimated earlier. However, tne forecasts at each point 
have to be updated as soon as the sample available. 

It is rather cumbersome to do in the above form. Hence, a different 
approach is required. 

Instead of writing the original equation as 

+ Vt-2 + ••• + V.-P * 't - 


we 


will express in a weighted infinite sum of e^ , > ••• 


62 



*t ' 2 Vt-J ' 

* JmO •’ •’ 

Where the b *s can be e^q>res8ed as 

4 # 


•> 0 ' » 

S ■ Vo 
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"j “ Vj-1 * Vj-2 ^ Vj-P • " 


The BiniBiui-Bean-square-error forecast Is 


X . = S b e 

t+ilt j tte-j 


(4) 


This foni lends itself easily to updating. This can be denonstrated 

Vtim 


after arrives. We may write 


and hence 


*t+ii t+i ■ 




(5) 


Thus, at time t + 1 , the forecasts for lead lengths up to / - 1 
can be easily obtained from the predicted values at time t for lead 
lengths up to / by using £q. (5) and by recognizing 

®t+l “ ^t+1 " ^t+ljt • 

This provides a convenient scheme for updating forecasts uf rxare 
data become available* 

The variance of the forecast error may be easily obtained from 
£q. (4)s Since the i-step ahead forecast error at time t i given by 
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i-1 

®t+i|t ' \+i ~ *t+i|t ' • 

The variance V(i) nay then be expressed as 

2 2 

V(i) = E b, (6) 

>0 J * 

This variance gives us a measure of the goodness of fit for the model. 
Since «e have assumed the e's to be normal, the conditional probability 
density function P(X^^^|X^, ... ) 'fs normal with mean V/|t 

and variance V(i) . Based on this information, we can obtain confidence 
intervals for the forecasts. For example, 95^ of the normal distribution 
mass lies within + 1.96 a of the mean where a is the standard deviation. 
Hence, + 1.96yv” is the 95^ confidence interval for the forecast, 

i.e., the probability that the actual value will lie in that range is 


.95. 


The previous formulation gives the forecasting procedure to predict 
values X samples ahead from any time t • In order to increase the 
lead time for forecasting, one has to increase the value of i . It is, 
however, evident from Eq. (6) that the error variance becomes larger 
as X increases and the confidence interval fer a given level of 
confidence will also increase accordingly. Another way to increase 
the lead time would be to increase the sampling interval and thus keep 
the value of £ as low as possible so that the level of performance 
of the predictor may be maintained. However, the sampling rate or 
sampling interval for a signal is determined by the bandwidth of the 
signal. If the signal is furnished in digital form, then one has 
little c loice in altering the sampling interval. One could, of course, 
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increase this interval by dropping every block of data points. In 
any event, the increase in saapling interval aeans reduction in the 
nuaber of data points in a given interval of tine. This reduction in 
data points leads to a higher variance for the estiaates. Besides, 
the nuaber of data points aay be so auch reduced, since the EEC signal 
is considered stationary only for a short period of tiB'>, that the 
estiaates are not statistically significant. For these reasons, we 
iCfu.Tu.iate the autoregressive aodel in a aore general fora as 

t I t-k 2 t-2k p t-pk t 

which, when k = 1 . reduces to the case discussed earlier. Th: ^ 

e<|uivalent to an autoregressive aodel for the tine series 




) 


T! e least-square estimates of the parameters can be obtained by a 
logical extension of the earlier results; viz.» 

“ 1 S 2 ®2^23 %S,p+l ^ “^12 

“iSa ^2^3 ^ %**3.p+l ~ ~^13 (7) 


“lS.p^-l ^ ®2S.p+l ®p%+l,p+l "^l,p+l 


where the redefined as 

i=l 


IJ 




l)k+l + ••• + V(J-l)k* 




Sj ^ *(i-l)k+l*(j-l)k+l ^ *•* ^ \-(j-l)k*n-(i-l)k ’ 

i 1 , J M • 

The least-square estimates are the solutions of this new set of equations. 
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The forecast equations are modifield in a similar manner to give 


*t-Hcijt ) 1 1 ''' S’^t4-k(i-2)| t '*’**• 



is demonstrated by our results* 

We developed a computer program to implement an algorithm for 
recursive estimation of parameters and forecast of future values by 
using the new modified scheme. The program was written in FORTRAN for 
a PDP-15 computer. As an example for illustration, we applied our 
scheme to real EEG data, which were recorded while the subjects with 
closed eyes were stimulated by stroboscopic flashes for 50 seconds at 
the rate of 10 flashes per second, then no stimulation for 50 seconds, 
then .'nother 50 secom's of stimulatirn, etc. The signals were sampled 
at an interval of 1,2 msec. Since we are mainly interested in the 
alpha component of the EEG, the data were filtered to obtain the 
components around the alpha range (7*13 Hz) through a transversal filter. 
The computer program reads ^his EEG data as its input. Other required 
information such as the number of sample points N for which the process 
is considered stationary, the order of the process P , the resampling 
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value K , the prediction lead length L and the window width NWND 

for updating the parasetera are also read in as input. (For efficiency, 

the parameters are not updated with each new data point, but only after 

NWND new data points have been read in.) A flow chart of the program 

is shown in Figure 33« As soon as the above variables and the initial 

N points are entered, the program computes the least-square estimates 

of the parameters by solving the set of linear equations (7). The error 
2 

variance a is also estimated* These estimates are then used to 
e 

generate minimum mean-square error forecasts up to L steps ahead; 
i.e., forecast of t + KL is obtained at time t • The variance of 
the forecast errors and the confidence interval for a certain significance 
level are also calculated. At this time, the next data point is read 
in; the residual is calculated as the difference between the actual 
value and the forecast; and the forecasts for the remaining points are 
updated by using £q. (5). A new forecast for the value L stops 
ahead is produced. This process is repeated until NWND new points 
have been exhausted. The procedure will repeat again starting with 
the recalculation of the parameter estimates. 

Results of the prediction for various lead times are shown in 
Figures 34 through 38. In order to compare the predicted waveforms with 
the actual ££G, we plotted the predicted waveforms and then superimposed 
the actual E£G on top of them. The order of the autoregressive process 
was chosen as p = 7 by examining the partial autocorrelation function 
of EEG data for k = 10 . The effective lead time in seconds is 
kiT where T is the sampling interval and i denotes the lead time 
in number of samples. In Figures 34 through 38, the waveforms were 
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Figure 33 A flowchart for the realization 
of the prediction scheme 
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Figure 34 Actual and predict'^ci wavelorm 
for lead time of 24 



Figure 35 Actual and predicted waveform 
for lead time of 48 msec. 
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Fitjurc 37 Actual and predicte'^ waveform 
lor lead time ol 96 Msec. 
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■ ij;ure Predict?on error variance 

vs. lead lime 
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plotted against the sample numbers and the lead tine i was indicated 
at the lower right comer. For instance^ in Figure 36 , i = 5 means 
a lead tiae of 10X5 X 1.2 = 60 msec., since k = 10 and T - 1.2 msec; 
hence, thr length in « .c«4ted for i = 5 represents 60 msec. The amplitude 
scale is relative. Tne solid curves represent the actual EEC and the 
dotted curves are the oredicted waveforms. In Figur. 34, we s^e that 
for a lead tine of 24 msec., the two curves practically coincide with 
each other. This indicates a prediction with little error. We progressive- 
ly increased the lead time from 48 msec, in Figure 35 to 120 msec, 
in Figure 38. It is observed from this series of figures that the 
coincidence of the predicted and the actual waveforms worsens. This 
indication ot increasing error as lead tine increases is expected. 

For a qua::! itative comparison, we computed the prediction error variance, 
which is us^d as a performance measure of the prediction scheme, for 

vaiious lead times. This forecast error variance is s'lown in Figure 39, 

2 

The vertical scale gives V(jl)/E(x^) which represents the prediction 
error variance relative to the variance of the signal to be preuicted. 
for example, for i = 5 (i.e., a ..ead time of 60 msec.) the error 
variance is l^t of the signal variance. This gives a measure of Low 
close the predicted values will be to the actual values. We can sec 
a glance from Figure 39 that the performance deteriorates as leaa 
time increases. In using this scheme, one can determine a maximum 
lead time f r any given t^^lerance level. For our purposes, this 
scheme is satisfa' 
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IV. CHARACTERIZATION OK SCAN PATTERNS 


One of the inportant aspects of the research done in this project 
has been the development of techniques and tools for the automatic 
analysis, characterization and display of scan patterns. In visual 
tasks, the sequence ol fixations used in scanning the visual target 
is called the ”scanpath", and we will refer to a "scan pattern" as a 
collection of information about the properties of one or more scan- 
paths recorded by one or several different observers* Naturally, 
to be able to dif lerent i*-te a superior scan pattern which results in 
superior visual memory from an inferior one, we must first be able to 
characterize a scanpath in some terms other than just a list of (X,Y) 
fixation coordinates. 

A graphical illustration of a sequence of the foveal fixations 
is shown in Figures *40 and 41. Figure 40 shows the outline drawing 
of a still life scene which was inspected by a subject with his scanpath 
superimposed- In ligure 41 'we have plotted a sketch o! the visual 
stimulus imoressed on the fovea for f ach fixation point shown in 
Figure 40, The sketches show a visual field of 6^ in diameter, a 
region which includes all of tJie lovea under practically any definition. 
These drawings, then, represent liT*e sequence ol snapshots which are 
the visual input to the ooserver who must a unified picture of 

the scene being viewed i rom the sequence of picture fragments shown in 
t ho sketches. 

One obser'^ation which has often been made is that the fixation 
poir.‘s tend to be grouped around certain features or areas of the 


visual image, and are relatively sparse elsewhere. Thus, we can measure 
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the nearness of one fixation point to another and assign a sinixarity 
measure to the tvo coints based upon this distance* The ability to 
group or c^u3ter a set of fixation points is an inportant step in the 
characterization of a scanpath, and several different techniques hcve 
been used to Measure point similarities and to assign points to clusters* 

The use of a "point sinilarity function" for clustering fixation 
poir^^s has been reported earlier and will not be reviewed here. 

We have also used two alternate algorithns for selecting cluster 
centers, ea^*> with its own particular advai^cages and disadvantages. 

These algorithas are known as "iSOI and "llininum-Spanning -Tree-Clustering" 
or MSTC. 

In the ISODATA oiethou, the number of clusters desired is given 

an ^^put *'ariable, and the algorithm partitions the fixation points 
into subsets such that the total distance between the fixation points 

in a and the cluster center is minimized for all clusters* 

The cluster center starting points are usually selected randomly, and 
the algorithm keeps moving the cluster centers until the above 
crlteixon has been reached. 

The MSTC algorithm automatically determine the number of clusters 
in the data and assigns fixation points to the clusters. Three 
variables given as input may be adjusted to c. .nge the criteria by 
which clusters ere seHcted. 

The chief advantages or disadvantages of these two algorithms are 
sui:.inarized below: 
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ISODATA 


(1) Required the desired number of clusters, which usually is not 
known. 

(2) Solutions often are not unique, but vary with starting conditions, 

(3) The iterations can require a substantial amount of computing, 

(4) Many points can be easily accommodated. 

MSTC 

(1) The solution is unique, and the number of clusters is automaiically 
determined, 

(2) The computation is quite fast. 

(3) Many points requires a large amount of storage. 

Our approach to characterizing scan patterns is based on 
the following two assumptions: 

(a) There are a discrete number of ’’fixation centers” in the scene 
being viewed and each fixation point in the scan can be assigned 

to one of the fixation centers. This process of assignment or 
clustering is done using the MSTC algorithm described above. The 
center of the clusters should be in close agreement with the 
tenters of the fixation. Due to the random nature of the fixation 
points, in any repetition (cycle) of the **canpath, the fixation 
points do not always coincide, 

(b) A scanpath then becomes ? sequence of transitions from one 
cluster center to another. Any saccade with starting anc' ending 
points in the same cluster is discarded. 
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Fron the set of all saccades leaving a given cluster center, we 


can find the transition or transitions which have the highest probability. 
These transtions are temed the **most-probable-saccades". The *'most- 
probable-scanpath** is then defined to be the set of all most-probaole- 
saccades* The nost-probable-scanpath depends on how the clusters are 
chosen, and is not necessarily a closed path through the cluster centers. 
Its usefulness is that it concisely sumnarizes the most iB 4 >ortant aspects 
of the scanpath ueed by the observer in viewing the given visual target* 

A scan pattern is some set of aost-probable-scanpaths for a given observer 
viewing the same ur different targets, or for several observers viewing 
the same target, etc* 

Figures 42 and 43 illustrate how these methods may be employed to 
reduce scanpath data to a more concise form* Figure 42 is a graph of a 
scanpath superimposed upon a line drawing representing the still life 
scene which the observer was viewing. The first recorded fixation 
(1) is in the center of the avocado. The last fixation (45) is in the 
m'ddle of the carrot. The viewing time required to create this 
scan totaled 20 seconds* F'igure 43 represents the same data after 
processing by the MSTC program* The program partitioned the fixation 
points into 11 subsets or cluster;; (using certain criteria supplied by 
the programmer) and then ccmou ed the most likely transitions from 
cluster to cluster. Clusters contain .ng only one fixation point were 
ignored. The resulting plot shows the most likely saccades bet>fceen areas 
containing large numbers oi fixations, and pres ts the ori^’-iual data 
in a highly condensed format. 

In ordc^ to compare one set of fixat * 3 or one scanpattem 
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Figure 43 Most probable transitions between cluster centers 


against another, criteria must be established which can be used to 
evaluate subtle differences in the following characteristics: 

(1) Total number of clusters - The MSTC algorithm partitions 
all the fixation points into some number of clusters, very 
rarely will the number of clusters in two different scans be 
the same. Rules must be established for eliminating or merging 
clusters which conplicate the analysis. For example, clusters 
containing a single point can usually be removed, 

(2) Similarity between clusters - The geometrical center of 
two clusters will usually be different, and yet it may seem 
obvious that both clusters are associated with or centered on 
some feature >f the picture, 

(3) The most probable saccadcs between clusters will often be 
different in two scans, even if the cluster centers are nearly 
alike. The similarity of two scans is Judged by their transition 
probabilities. 

The above criteria are dependent upon the type of eXw'^i'iment bein^ run 
and no fixed set of rules will be suggested here. Based on the above 
criteria and using statistical methods, we are able to show from our 
preliminary results that the scan patterns of the same visual material 
by the same subject who scored very high on Marks' Vi<:ual-*Meraory Task 
are consistent. 
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V. TECHNIQUES FOR THE PRESENTATION Ob VISUAL CUES 


The measurement and prediction of eye movements and related EEC 
potentials are necessary prerequisites for the control of visually 
displayed material which can enhance an observer’s perception or memory 
of a scene or visual environment. This section describes some means 
to couple the measurement of eye movements with the control of a 
computer-driven graphical aisplay. 

The task undertaken was to provide a facility for dynamically 
controlling the tovcal or peripheral field of view of an observer. Such 
a tacility would provide a flexible method ior studying interaction 
between eye position and changes in the visual field. Some of the ob- 
jectives of this research are: 

(a) to uevelop techniques lor dynamically controlling the visual 
field in humans without any restrictive mechanical attachment to 
thv* eye; 

(b^ to obtain a subjective Iceling for percep^^ion without use of 
the normal visual field; and 

(c) to quantitatively measure changes in the observer’s pevformunce 
(and scanpath characteristics) as a function ol field size and type. 
The system works as lollows: At the start ol each refresh of the 

display (30 times per second), the position of the eye is measured and 
the corresponding point on the display is computed. Parameters within 
the program determine what portion of the visual licld will be displayed — 
lovea only, periphery only, and the size the field in either case. 


Then, each point in the display list is chew-Ked, and il it is within the 




required fieldi it is displayed. Figures 44 and 45 depict the stimulus 
for the two modes of display. The box (which was used instead of a 
circle to simplify computations) remains centered on the fovea, no matter 
where on the screen the obsetver looks. 

The visual scenes used in these experiments were outline drawings 
of simple, still scenes. No text or highly detailed pictures were 
involved. A new scene could be entered iuto the conq>uter through use 
of a graf/pen digitizer. Once entered, the points in the picture were 
stored by X,Y position* and a display list was created for later use 
by the real-time interactive program. 

After adjusting the eye tracker and calibrating the instrument, 
the observer would be presented with a field of view which could be 
either a fovea-only presentation or a periphery-only presentation. 

The size of the field (width of box boundary) was also adjustable, from 
less than one degree to the maximum width of the display . Use of this 
program has produced, so far, these results; 

Impaired perception for fovea-only vision - If or ly a two or three degree- 
wide foveal field is displayed (when the display Itself spans twenty 
or more degrees), the observer finds it difficult, if not impossible, to 
perr:eive the display. What the observer seec are short line ;^egments, 
and sometimes Junctions between line segment:: The only way to perceive 

an object is to consciously follow the contours and mentally reconstruct 
the path followed. Simple objects take several minutes to detect. few 
observers get a complete Idci- of tha entire scei^e. Far more use of 
peripheral visual inform. .Mon ^ made than was anticipated. Large 
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saccades must be forced, and the observer becomes very conscious of having 


to manipulate his own eye movements. 

Perception with peripheral vision - If the foveal region is blanked, 
little perceptual loss occurs until the window size approaches roughly 
10 , that is, the scene is still perceived even when the window is 
Quite large. Our stimuli did not include text or other material where 
high visual acuity was essential. This result is somewhat surprising 
in view of the general assumption of the importance of foveal-high 
acuity field. The peripheral field apparently makes an important 
con ributron to the overall percent ion of a scene, even the visual 
acuity nay be quite low in the outer regions. 

Iriportance of closed contours - If the fovea-only visua! field is being 
displayed, little improvement in perception of objects within the 
field occurs uncil the window size is large enough to include the complete, 
closed contour of an object. The absolute size of the foveal region seems 
to be less important than the fact that complete objects are visible, 
at least ^ r the line drawings we used. This result supports another 
observation we have made, which is that fixations tend to be centered 
on the main surfaces of a simple object, and are not as often bound on 
the edges or corners of the object. 

Appearance of a phantom window - After about ten seconds of viewing, and 
especially for the fovea-only condition, a phantom window with pronounced 
grey-black edges appears. The subject sees a grey background with a 
black window cut in the center, through which the white lines may be 
viewed. The window, of course, is fixated on his eye lik', an alter image. 
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Two causes for the vindow are evident. Only the foveal region receives 


stimulation, and the rods and cones adapt to a higher ambient light 
level than in the periphery, which adapts to "visual grey". Also, the 
sharp truncation of many contours along a straight line reinforces the 
perception of an edge. Similar effects have been noted before by Yarbus. 

These preliminary experiments with an eye-movement controlled 
visual display have proved quite interesting and useful. The next step 
is to incorporate the predictive algorithms discussed in other previous 
sections of this report to present the visual cues at the desired 
locations of the visual field at the desired moments. 
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VI. CONCIXSIONS 


Our aim has been to develop and use bioc-'bernel ic techniques for the 
enhancement of visual memory. To this end, we have desij^icd a closely- 
coupled man-machine system for implementation, have developed and 

completed the models for saccadic eye irovements and KEG signals in this 
man-machine system. The models are used for monitoring and prediction of 
eye positions and brain states, ^’ith the continuously up-dated information 
concerning the eye position and brain state for adjusting the stimulus 
parameters and the monitoring and prediction schemes, we can guide the 
eyes to fixate at the specified locations of the visual field at the 
specified instants oi time through a technique we developed. If these 
specified locations and instants are the optimally required for superior 
visual memory, then the vividness and persistence of the desired 
after-image will be enhanced. ^Ve have approached the problem of 
determination of these optimal locatii»ns and time instants by monitoring 
e> e movements and KEG signals through the models which we have developed. 

Through the KEG model, we have obtained a bcvter assessment of EEC’s 
role as the timing mechauiSm for visual information acquisition and 
processing. *'e have also shown that the scan pattern is more consistent for 
individuals with good visual memory (as scored by :.!arks* Visual -Memory 
Task) than those w'iih poor visual memory. This comparison is made possible 
by our method of characterization of scan patterns. With these results, 
it IS possible to determine the optimal locations in the visual field 
and the optimal lime instants lor presenting visual stimuli. W'c conclude 
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that the approach for visual Beaory enhancement as proposed is a feasible 
one. At the termination of this project, we have implemented all of 
the parts shown in the closely-coupled man-machine system but the 
feedback path. 

Sipiificant accomplishments made during this research have been 
published or presented in international or national scientific conferences 
and two Ph.D. dissertations are near completion. 
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APPENDIX A 


EI^CTROWIC INTERFACE SYSTEM 

shall describe here the interface systen for eye-movement data 
acquisition only. We are presently usirg a double Purkinje image ^ye 
tracker developed at Stanford Research Institute (SRI) for measuring 
eye movements. It relies on measuring the motion of the reflections 
from the front surface of the cornea and from the back surface of the 
lens of the eye (the first and fourth Purkinje imares). The instrument 
is discussed in an article: ‘^Accurate two-dimensional eye tracker 
using first r,.nd fourth Purkinje images", by T. X. Comsweet ard H. D. 
Crane in the Journal of the Optical Society of America , vol. 63, no. 8, 
pp. 921-928, August, 1973. Figures A.l and A. 2 show the set up of this 
eye tracker. 

The first ard fourth Purkinje images are generated by positioning 
a narrow beam of infrared light on the subject’s pupil. The image of 
the eye and its attendant Purkinje image reflect off the infrared mirror 
in front of the subject, through two large collimating lenses and on to 
a movable mirror. A four-quadrant pholodetector senses the position of 
the first Purkinje image. Signals from these four quadrants are used 
to drive two nigh-speed servo motors to detect the movable mirror m 
altitude and azimuth. The function of this servo system is to position 
the image of corneal reflection when the eye moves so that the reflection 
is always in the same position; thus, a stable reference of the eye is 
provided. As the eye moves, the image reflected by the movable mirror 
will remain stationary, A second optical system in tandem consisting of 
ai. other movable mirror and quadrant photodetector tracks the fourth 
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Purkinje image and measures its movement relative to the first image. 

The two-dimensional motion of this image derived from the position of 
this mirror system is used as the output signal of the subject's eye 
movements. The system is designed to measure eye movements with an 
accuracy of up to two minutes of arc. The overall system has a flat 
frequency response of up to 100 Hz. 

To assure the proper use of the eye tracker requires elaborate 
procedures for alignment of subjects and calibration of the eye tracker. 
The calibration is done through a computer program (MAP) we developed. 
This calibration program serves tc : 

(1) provide a way of relating the output voltages of the instrument 

to the stimulus positions; 

(2) give an estimate of the size of the visu^*l field over which 

the instrument is able to trac ; the eye movements; and 

(3) indicate whether the eye tracker is correctly adjusted. 

It has been a major problem to relate the measured eye positions 
to the actual locations on the scene. This difficulty arises mainly 
from the distortion introduced in measuring eye movements. To alleviate 
this difficulty, v,e have developed and implemented a scheme which maps 
automatically the eye fixations onto the scene without distortion. This 
is achieved by expressing both the fixations and the line drawing of 
the inspected scene in terms of visual degrees relative to a calibration 
slide. Since both are on the same coordinate system, they can be easily 
superimposed without any fear of distortion. 

Our scheme consists of the following computer programs: MAP, 

MXKIL, OVLAY, and PFIX. MAP produces a calibration file relating 
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eye tracker voltage with the visual angle subtended by the eye. FIXFIL 
analyzes the data collected by the DIGIT program. It reduces the data 
into fixations and» with the aid of the calibration file produced by 
MAP, it translates the location of these fixations into units of visuPl 
angle. The origin of this visual coordinate system xs assumed to be 
the center point of the calibration slide. OVIAY is a variation of the 
DRAW program ibhich uses the graf/pen to digitize pictures. OVLAY is 
used to digitize an outline of the inspected scene on a slide. Some 
additional data are added to the file to enable it to be eventually 
translated into units of visual angle with the exact coordinate system 
used by FIXFIL. PFIX plots the fixations and superimpose the outline of 
the scene. This can easily be done since the fixation locations and the 
outline are expressed in the same visual angle coordinate system. 

To illustrate the effectiveness of this scheme, we show an example 
in Figures A. 3 and A. 4. The observer was asked to fixate at those 
circled points of a scene as shown in Figure A. 3. The eye-movement data 
taken were then processed by the mapping scheme. The result was plotted 
as shown in Figure A,4 , It is seen that the measured eye fixations 
correspond very closely to those circled points in Figure A. 2 in the 
right sequence. 

It should be emphasized that the process described is fully automatic 
and will compensate for different subjects and any of the permissible 
variations in the experimental set up such as size of stimulus, distance 
between the screen and the observer, etc. With this system, we are 
capable of processing massive amounts of eye-movement data with re ative 
ease. 
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FlRure A. 3 Requested flxntion points 
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Figure A. 4 Responses to requested fixation points 
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